Method and system for expressing affective state in communication by tele- 
phone 



The invention relates to a method and system for expressing the affective state of a 
caller and/or called party to the conversation partner in communication by telephone 
5 where the caller and called party send messages to each other. 

When talking, people use a lot of expressions and gestures which reflect their emo- 
tions. The gestures and expressions complement spoken messages and may some- 
times reveal more of the contents of the message than the words uttered by the 
interlocutors. There are also situations in which a speaker cannot or will not express 

1 0 his feelings in words but waits or hopes that the listener is able to infer the state of 
mind of the speaker and guide the conversation accordingly in the appropriate 
direction. In a normal conversation in which the speaker and listener are near each 
other and see each other all the time, the expressions and gestures of the interlocu- 
tors are easily perceived. Often the interlocutors are able, even before a single word 

1 5 has been uttered, to "read" the state of mind of their conversation partners in their 
faces and are thus able to assume an appropriate preconception of the forthcoming 
message. For example, a happily grinning friend may be expected to utter a humor- 
ous greeting as his first words. Similarly, when grief or worries are reflected in a 
friend's face, the forthcoming message may be expected to contain sadder news. 

20 The situation is not so simple when the interlocutors do not see each other. This is 
the case in telephone conversations, for instance. If the parties do not explicitly tell 
each other their emotions, they may infer them only on the basis of the tones of 
voice heard on the phone. Especially in the case of cellular phones, all voice signals 
uttered on the phone will not be conveyed to the listener because of limitations of 

25 the technology used so that it is difficult to estimate the affective state of the con- 
versation partner. Poor audibility and interference in the calls naturally make such 
estimation more difficult still. As the body language based on expressions and ges- 
tures is completely left out, the nature of the conversation situation changes. Part of 
the intended content of the messages may then be not understood. Incorrect estima- 

30 tion of the affective state of a conversation partner may thus lead to misunder- 
standings, unintentional insults and embarrassing situations. 

An object of the invention is to provide a new method and system to improve the 
intelligibility of communication by telephone as well as enhance the chances of the 
interlocutors to express their affective states in a telephone conversation. 
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The method and system according to the invention are characterized by that which 
is specified in the independent claims. Advantageous embodiments of the invention 
are presented in the dependent claims. 

The basic idea of the invention is that the content and intelligibility of communica- 
5 tion by telephone are improved by means of music. Musical compositions are typi- 
cally works that create in a listener an association with a certain emotion. A person 
will listen to different kinds of music in different states of mind. A happy person 
will usually listen to fast, sprightly music and, conversely, a sad person usually feels 
that heavier, more restful music will better suit his state of mind. In this invention, a 

10 suitable musical composition reflecting the state of mind of the sender of a message 
is played in the background of the message conveyed on the phone. Thus the 
speaker is able to communicate his affective state to the conversation partner with- 
out being compelled to tell it explicitly in words. Similarly, the music playing 
behind the voice of the speaker will immediately give the listener an idea of the 

1 5 state of mind of the speaker so that the listener will be able to adopt an appropriate 
attitude in the situation. When the recipient of the message understands the state of 
mind of the caller, he is able to better interpret the things said by the caller. 

An advantage of the invention is that it increases the usefulness of the cellular 
phone since part of the communication situations that earlier required personal 
20 intercourse can, by means of the invention, now be handled by telephone. 

Another advantage of the invention is that it adds to the intelligibility of communi- 
cation by telephone, thereby reducing the risk of misunderstandings and uninten- 
tional insults. 

A further advantage of the invention is that it shortens phone calls as the music 
25 complements spoken messages and makes it easier to understand the message. 
Shorter phone calls mean smaller phone bills, thus bringing savings to the user of 
the phone. 

A still further advantage of the invention is that it is technically feasible to 
implement. 

30 The invention is below described in detail. In the description, reference is made to 
the accompanying drawings in which 

Fig. 1 shows by way of example a schematic of an advantageous embodiment 
of a menu used in the selection of background music, 
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Fig. 2 shows by way of example a schematic of an advantageous automatic 
state of mind recognizer, 

Fig. 3 shows by way of example a schematic of an advantageous embodiment 
of the invention, 

5 Fig. 4 shows by way of example a schematic of a second advantageous 
embodiment of the invention, 

Fig. 5 shows by way of example a schematic of a third advantageous embodi- 
ment of the invention, and 

Fig. 6 shows by way of example a schematic of a fourth advantageous 
10 embodiment of the invention. 

In the method according to the invention, the background music for the message is 
selected using a menu function implemented in the cellular phone. Suitable back- 
ground music is selected from the menu either manually or by means of an auto- 
matic state of mind recognizer. Manual selection in this description refers to all 

15 selection methods that require deliberate actions of the user of the phone. One 
advantageous manual selection method is to browse the menu using the phone's 
keypad and choose a suitable emotion or musical composition. Another advanta- 
geous manual selection method is to fit a switch in the case of the cellular phone 
that reacts to squeezing force. In this embodiment the menu can be browsed by 

20 squeezing the switch at different levels of force so that icons representing different 
emotions are presented on the phone's display. As soon as the desired emotion is 
displayed, it is selected by pressing a key. Yet another advantageous manual selec- 
tion method is to have a software-based speech recognition function in the cellular 
phone. In this case the user chooses a musical composition representing an emotion 

25 to be played as background music by pressing the speech recognition key and at the 
same time uttering the code of an emotion or musical composition. 

Manual selection of an affective state is advantageously carried out in connection 
with the starting of a call or prior to it. The user of the phone may e.g. pre-select an 
emotion representing his state of mind and make it a default. Then, the background 
30 music of all messages sent after that selection will comprise the musical composi- 
tion according to the default value unless the affective state is changed in connec- 
tion with call set-up. Moreover, affective states or musical compositions may be 
selected in advance for different user groups or profiles separately. In that case, the 
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musical composition played in the background of the message is changed on the 
basis of the user group to which the recipient of the message belongs. 

Fig. 1 shows by way of example a schematic of an advantageous embodiment of a 
menu 28 used in the selection of the background music. The musical compositions 
5 10a, 10b,. . ., lOn are stored in electric form, advantageously as midi or mp3 files, in 
a directory 30 which may be located in the cellular phone of the caller 32 or the 
called party 34 or in the operator's 36 files (Figs. 3, 4, 5, 6). A suitable piece of 
music is searched in the directory by means of the menu 28 in the cellular phone. 
The menu contains a set of names of different affective states 20a, 20b,..., 20n. For 
10 example, affective state 20b may represent joy, and state 20n depression. For each 

t! affective state in the menu 28 the directory 30 contains at least one musical compo- 

sition 10a, 10b,..., lOn which reflects the affective state in question. The names 
11a, lib,..., lln of these musical compositions are also shown in the menu 28. 
Naturally the directory may also contain several musical compositions representing 

CP 15 one and the same affective state. The user of the phone selects in the menu the 
affective state 20a, 20b,..., 20n which in his opinion best represents his current state 
of mind and takes it into use in a manner described above. The background music in 
a call made by the user is then the musical composition 10a, 10b,..., lOn according 
to the selection made. If several musical compositions are associated with the 
20 selected affective state, a system programmed in the menu functions chooses one of 
them. Naturally the user of the phone may also select the background music direct 
by the name 11a, lib,..., 1 In of the musical composition 10a, 10b,..., lOn whereby 
the selected musical composition is played as background music. 

Fig. 2 shows by way of example a schematic of an advantageous automatic state of 
25 mind recognizer 40. The automatic state of mind recognizer is an electronic system 
which can be connected to a cellular phone for estimating the affective state of the 
user of the phone and which comprises an electromyogram (EMG) sensor 42, respi- 
ration sensor 44, galvanic skin response (GSR) sensor 46, and a blood volume pres- 
sure (BVP) sensor 48 as well as a decoding device 50 which interprets the signals 
30 transmitted by the sensors. Data gathered by the sensors are transferred to the de- 
coding device advantageously using wireless technology. The decoding device is 
attached to the cellular phone 33 in either a fixed or detachable manner. On the 
basis of the data received from the sensors the automatic state of mind recognizer 
makes an estimation of the state of mind of the user of the phone and selects 
35 accordingly an appropriate affective state, whereby a musical composition repre- 
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senting the affective state is automatically set as the background music for the 
message sent. 

The sensors in the automatic state of mind recognizer 40 are placed on the surface 
of the cellular phone user's body so that certain bodily functions and changes in 
5 them can be measured by the sensors. The EMG sensor 42 comprises a number of 
small electrodes attached around the neck and shoulder muscles of the user. It has 
been found that these muscles become tense when a person is angry, for example. 
The EMG sensor measures the tension of these muscles so that the measurement 
results indicate when the person is feeling anger. 

10 The respiration sensor 44 is an elastic instrument fitted tightly around the chest of a 
person so that it stretches in accordance with the movements of the person's chest. 
The sensor transmits a continuous signal which varies according to the stretching of 
the sensor. Thus the sensor can be used to measure a person's respiration rate and 
breathing depth whereby it can be deduced e.g. whether the person is in a calm or 

1 5 excited state of mind. The respiration sensor may be a separate band-like part or it 
may be integrated in a tight-fitting piece of clothing covering the upper body of the 
user. 

The GSR sensor 46 comprises two small electrodes placed against a person's skin 
surface at a distance from one another. Electric current flowing between the elec- 
20 trodes can be used to measure the skin's electric conductivity which has been found 
to vary according to the feeling of anxiety experienced by the person. The elec- 
trodes, which advantageously are silver chloride electrodes, may be advantageously 
attached to the inner surface of a glove so that they are automatically placed against 
the surface of a person's skin always when the person is wearing the gloves. 

25 The BVP sensor 48 is used to measure the blood flow rate in a person's vascular 
system. An increase in the blood flow rate often indicates fear and anxiety. The 
BVP sensor emits infrared radiation onto the skin surface and measures the amount 
of reflected radiation whereby the person's blood flow rate can be estimated. The 
BVP sensor may be advantageously attached on the finger tip of a glove. 

30 The decoding device 50 is an electronic device which receives the signals coming 
from the sensors and uses them to make an estimation of a person's state of mind. 
Based on the estimation, the decoding device selects the most suitable of the affec- 
tive states available so that a musical composition representing that particular affec- 
tive state is automatically set as the background music for the message to be sent, 



the message being advantageously a voice message. A cellular phone may be 
programmed so as to include a function which requests the user of the phone to 
confirm the affective state selected by the decoding device e.g. by pressing a key on 
the phone. The user of the phone may also choose whether the phone's automatic 
state of mind recognizer is on or off. The automatic state of mind recognizer may 
also be used together with the manual affective state selection. For example, at the 
beginning of a call the caller may set his affective state manually, and during the 
call the automatic state of mind recognizer monitors the caller's state of mind and 
changes the affective state selection when needed. 

In addition to the sensors mentioned above the automatic state of mind recognizer 
may be connected to other biosignal sensors, recognition devices or cameras in 
order to measure the bodily functions of the user of the phone so that the state of 
mind of the user can be estimated on the basis of the data produced by such devices. 
The state of mind may be estimated on the basis of the voice or facial expressions of 
the speaker, for example. 

Fig. 3 shows by way of example an advantageous embodiment of the method 
according to the invention. In this embodiment, the directory 30 containing the 
musical compositions 10 to be used as background music for messages 60, is 
located in the cellular phone 33 of the caller. When starting the call, the caller first 
determines his state of mind 52. He may do it independently based on his own 
judgment, or it may be done by the automatic state of mind recognizer 40 depicted 
in Fig. 2. On the basis of the state of mind, a suitable affective state 20a,..., 20n or 
musical composition 10a, 10b,..., lOn is selected from a menu 28. Based on address 
information provided by the menu, the right musical composition is fetched from 
the directory 30. After that, the caller makes a call 54 to a person. Advantageously 
the call is made by entering on the caller's 32 cellular phone 33 the phone number 
of the cellular phone 35 of the called party 34. The call is connected 56 by an 
operator 36, after which the called party 34 answers 58 the phone. Now the caller 
and called party are in communication connection with each other and can send 
messages 60 to one another by means of the phone. Messages transmitted on the 
phone may advantageously be voice messages or they may contain e.g. images or 
text in addition to or instead of speech. The voice of the caller can be heard on the 
cellular phone 35 of the called party, and music selected by the caller can be heard 
in the background of the voice. In this advantageous embodiment of the invention 
the sound of the musical composition is transferred, like speech, on an audio 
channel from the cellular phone 33 of the caller to the cellular phone 35 of the 
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called party. The call is terminated 62 normally by hanging up one of the phones 
whereby messages and music cease to be transferred between the phones. 

Fig. 4 shows by way of example a second advantageous embodiment of the method 
according to the invention. In this embodiment the directory 30 containing musical 
5 compositions is located in the cellular phone 35 of the called party 34. When start- 
ing a call, the caller 32 determines his state of mind 52 and selects from the menu 
28 the appropriate affective state or musical composition, whereby an identifier 64 
for the selected affective state or musical composition is stored in the phone's 
memory. After that, the caller makes a call 54 to a person and the operator 36 con- 
10 nects the call 56. At this stage the identifier 64 for the affective state or musical 
composition selected by the caller is transferred via the operator to the cellular 
phone 35 of the called party 34. On the basis of the identifier a suitable musical 
composition 10a, 10b, lOn is searched in the directory 30 in the cellular phone 
of the called party and played on his phone when he answers 58 the phone. 

15 In this advantageous embodiment of the invention, only the identifier of the musical 
composition 64, in addition to speech, is transferred from the cellular phone of the 
caller to the phone of the called party where the said identifier is used to search a 
suitable musical composition in the directory in the phone. The identifier may com- 
prise the name 11a,..., 1 In of a certain musical composition or just the name of an 

20 affective state 20a,. . ., 20n representing the state of mind of the caller. The voice of 
the caller is transferred normally on an audio channel from the cellular phone of the 
caller to the phone of the called party, and the called party will hear the music rep- 
resenting the state of mind of the caller in the background. The call is terminated 62 
normally by hanging up one of the phones whereby also the music ceases to be 

25 played on the phone of the called party. 

Fig. 5 shows by way of example a third advantageous embodiment of the method 
according to the invention. In this embodiment, the directory 30 containing musical 
compositions is located in the operator's 36 equipment. When starting a call, the 
caller 32 determines his state of mind 52 and selects in the menu 28 the name of an 

30 affective state or musical composition, whereby a corresponding identifier 64 is 
stored in the phone's memory. After that, the caller makes a call 54 to a person and 
the operator 36 connects the call 56. At this stage the identifier 64 is transferred 
from the cellular phone of the caller to the operator 36, and on the basis of the iden- 
tifier a suitable musical composition 10a, 10b, lOn is searched in the directory 

35 30 in the operator's equipment. When the called party answers the phone, the 
operator adds the signals of the musical composition onto the same channel with the 
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message 60 sent by the caller. Thus the called party hears the music on his phone in 
the background of the message sent by the caller. The call is terminated normally by 
hanging up one of the phones whereby also the music ceases to be played. 

Fig. 6 shows by way of example a fourth advantageous embodiment of the method 
5 according to the invention. In this embodiment, the directory 30 containing the 
musical compositions 10 to be used as background music is located in the cellular 
phone 33 of the caller. The musical compositions are stored in the directory as files 
80 which may be playable audio files, advantageously midi or mp3 files, or data 
files containing the notes of the musical compositions. When starting the call, the 

10 caller first determines his state of mind 52 and, based on that, selects in a menu 28 
an affective state 20a,..., 20n or musical composition 10a, 10b,..., lOn representing 
his state of mind. Based on address information provided by the menu, the file 80 of 
the musical composition is searched in the directory 30. After that, the caller makes 
a call 54 to a person, and the operator connects the call 56. When the call is ready, 

15 the file 80 is transferred in its entirety through the operator to the cellular phone 35 
of the called party 34. As the called party answers the phone, the music 10 con- 
tained in the file 80 starts to play essentially simultaneously with the voice heard on 
the phone. The called party can choose whether to listen to the music on his cellular 
phone or on a separate sound reproduction apparatus 82, like e.g. loudspeakers, 

20 connected to the cellular phone. 

In this advantageous embodiment of the invention there is transferred from the 
cellular phone of the caller to the cellular phone of the called party, in addition to 
speech, an electric file 80 which contains the selected musical composition as a 
sound recording, or the note data needed to reproduce the musical composition. The 

25 called party can choose the device on which he wants to play the music. Thus the 
cellular phone may serve purely as a file receiving device from which the file is 
transferred to a selected sound reproducing apparatus 82 such as a home stereo 
system of the called party. Advantageously the cellular phone of the called party 
includes a function by means of which the called party may in advance direct all 

30 music data files arriving in connection with calls to be played on a certain device 
chosen by him. The voice of the caller is transferred normally on an audio channel 
from the cellular phone of the caller to the phone of the called party, and simultane- 
ously with the voice a musical composition describing the affective state of the 
caller is played on a device selected by the called party. The call is terminated 62 

35 normally by hanging up either one of the phones whereby also the music stops. 



Figs. 3, 4, 5 and 6 illustrate advantageous embodiments of the method according to 
the invention. These embodiments differ from each other only in the technical and 
software implementation of the method. All solutions are technically feasible and 
require only minor hardware and/or software changes in the cellular phones and/or 
operator equipment. The transfer of identifiers or files between cellular phones and 
operator in conjunction with call establishment can be realized using a known 
signalling technique widely used in cellular phones. Advantageously the identifiers 
and files may be transferred using the UUS (User-to-User) signalling technique 
according to the ETSI GSM 4.87 V7.0.0 specification. From the point of view of 
the user of the phone there are no significant differences between the embodiments 
described, for the visible functions associated with the selection of the affective 
state can be carried out substantially in a like manner in all embodiments. 

In Figs. 3, 4, 5 and 6 it is presented how a musical composition is associated as 
background music with a message sent by the caller 32. Naturally, telephone com- 
munication is bi-directional and, in accordance with the principles described above, 
a musical composition can be set to play in the background of a message sent by the 
called party 34 as well, so that the caller 32 hears it on his cellular phone. It is 
obvious that phones can be used to send not only spoken messages but other kinds 
of messages, too. The messages may contain in addition to or instead of speech also 
images or text, for example. The method according to the invention can be used also 
in such telephone communication in order to express affective states of the commu- 
nicating parties. 

The method according to the invention can be used in calls of all types. An example 
of such everyday calls between acquaintances could be a call made by a teenage girl 
to her friend. If the caller is feeling depressed or a little sad, she may select a sad 
piece of music as background music for the call. The called party will hear the 
music in the background and know that the caller is sad. The called party may select 
a cheerful piece of music to be played in the background of her voice and guide the 
conversation towards happier topics and thus cheer the caller up. 

By means of the method according to the invention the user of the phone may also 
communicate his or her situation to the caller. For example, a person answering a 
phone may be in a great hurry and doesn't have time to be on the phone for a long 
time. He may then select as background music a fast-paced composition which 
plays in the background always when answers the phone. The background music 
discreetly tells the caller that the called party is in a hurry at the moment so that the 
caller knows that he should speak briefly. 
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Yet another example of using the method according to the invention is the commu- 
nication of sad tidings by telephone. When the caller wants to tell his acquaintances 
or friends about the death of a friend or relative, for instance, he may choose as 
background music a hymn or a peaceful classical composition. The called party will 
5 hear the music when answering the call and immediately understand that the caller 
has some serious and sad news to tell. This way he is able to adopt a sympathetic 
attitude right away, making it easier for the caller to tell the sad news. 

Above it was described a few possible situations in which the method according to 
the invention could be used. Naturally, the method according to the invention may 
10 be used in calls of other kinds, too. Pieces of music may be used in the background 
even if they are not used to express a particular affective state. For example, the 
user of the phone may select as background music a composition performed by his 
favourite singer or group and thus communicate to his conversation partners some- 
thing about his personality. 

15 In an advantageous embodiment of the method according to the invention a musical 
composition is set to be played in the background of a text message sent by cellular 
phone. GSM phones, for example, include a so-called SMS (Short Message 
Service) feature enabling the sending of text messages between phones. The GSM 
standard gives manufacturers a choice to add proprietary data at the beginning of 

20 the text message. Thus the sender of a text message may add to this so-called User- 
Data-Header an identifier referring to a piece of music or affective state. Another 
advantageous method is to add the identifier of an affective state or musical compo- 
sition to the beginning of the text message in the form of text and separate it from 
the rest of the text by special symbols by means of which the receiving apparatus 

25 will know that the identifier represents information required for indicating a piece 
of music or an affective state. The identifier in question is transferred together with 
the text message to the cellular phone of the called party, whereby the correct musi- 
cal composition is searched in the directory in the cellular phone of the called party 
and played when the user of phone reads the text message. Thus a bride and groom, 

30 for example, may announce their forthcoming wedding by sending a text message to 
their friends and relatives and choose a wedding march to play in the background of 
the message. This way the recipient of the text message, while reading the message, 
will hear a musical composition in the background complementing the message. 

Above it was described some advantageous embodiments of the method according 
35 to the invention. The present invention finds particular utility in cellular phones but 
it can also be used, where applicable, in phones utilizing the traditional wired net- 
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work. The invention is also applicable when one of the phones of the conversing 
parties is a phone belonging to a wired network and the other a cellular phone. 

The invention is not limited to the solutions described above but the inventional 
idea can be applied in numerous ways within the limits defined by the claims 
attached hereto. 



